generalize crate for multi-device PCIe passthrough by cheese-head · Pull Request #1573 · NVIDIA/OpenShell

cheese-head · 2026-05-26T18:26:29Z

Summary

Generalize openshell-vfio beyond GPU-only / single-device passthrough so it can serve as the binding/validation primitive layer the VM driver needs to implement RFC-0004's resource_requirements model. Adds atomic IOMMU-group binding, dry-run validation for ValidateSandboxCreate-style paths, class-agnostic device enumeration, and a correctness fix for partially-failed binds. Purely additive plus one bug fix; consumer crates are unchanged.

Related Issue

Foundational for RFC-0004 sandbox resource requirements (#1360). Unblocks multi-device-per-sandbox passthrough that the existing single-device API could not express when devices shared an IOMMU group (consumer GPUs + HDA + USB-C, multi-PF NICs, devices behind ACS-deficient PCIe switches).

Changes

Add prepare_pci_group_for_passthrough / release_pci_group_from_passthrough for atomic bind/release of multiple PCIe devices sharing one IOMMU group. Rollback only restores devices newly bound by the call, so it does not steal bindings owned by other guards.
Add validate_pci_for_passthrough / validate_pci_group_for_passthrough as dry-run pre-flight checks for ValidateSandboxCreate-style paths. Performs every structural and IOMMU-peer check without touching driver_override or any other kernel state. prepare_* now delegates to its validate counterpart to keep the two in lockstep.
Add probe_host_vfio_candidates(sysfs, vendor_filter) for vendor-filtered, class-agnostic enumeration of passthrough-eligible PCI devices so consumers can advertise DeviceClassCapability for arbitrary classes (GPUs, NICs, VFs) instead of being limited to probe_host_nvidia_vfio_readiness.
Add PciBindGuard::companion_bdfs() accessor for consumer-side persistence of grouped bindings (crash-recovery state, status reporting).
Add VfioError::GroupMismatch and VfioError::EmptyGroup for typed validation responses.
Fix bind_device_to_vfio to clear driver_override and re-probe the host driver on drivers_probe failure and on post-probe polling timeout. Previously a failed bind could leave the device wedged with driver_override="vfio-pci" pinned on disk, causing the next probe event to silently re-bind to vfio-pci.

Testing

mise run pre-commit passes
cargo test -p openshell-vfio passes (52/52, up from 32)
cargo clippy -p openshell-vfio --all-targets -- -D warnings clean
cargo check -p openshell-driver-vm clean (consumer crate compiles unchanged)
Unit tests added (20 new), covering: atomic group bind happy path, rollback safety when a pre-bound peer must not be stolen, rejection of undeclared peers / mixed groups / duplicates / missing devices / empty slices, batch release continues after per-device errors, dry-run validators write no kernel state, and vendor-filtered probing including skip-on-no-IOMMU-group.
E2E tests added/updated (N/A — primitive crate, exercised end-to-end by consumer crates which are out of scope for this PR)

Checklist

Follows Conventional Commits
Commits are signed off (DCO)
Architecture docs updated (if applicable) — N/A, no public surface in docs/ covers openshell-vfio today

Signed-off-by: Patrick Riel <priel@nvidia.com>

copy-pr-bot · 2026-05-26T18:26:33Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

drew · 2026-05-26T20:58:33Z

@elezar adding this for your review as well since it's adjacent to gpu work

…ng restart reconciliation, without rebinding or mutating sysfs. Signed-off-by: Patrick Riel <priel@nvidia.com>

elezar · 2026-05-27T14:15:20Z

+    let vendor = read_sysfs_trimmed(&dev_dir.join("vendor"))?;
+    if vendor != NVIDIA_VENDOR_ID {
+        return Err(VfioError::NotNvidia {
+            bdf: bdf.to_string(),
+            vendor,
+        });
+    }


Question: Is this code NVIDIA-specific? If so, we may want to update the function name.

Signed-off-by: Patrick Riel <priel@nvidia.com>

Signed-off-by: Evan Lezar <elezar@nvidia.com>

elezar · 2026-05-28T12:07:08Z

I worked through this locally with Codex, and after Codex called out some
possible issues around the VFIO reference counts, we ended up with a small
refactoring introducing a VfioIdRegistry to encapsulate the refcounting logic.

The bug is in the cached-ID release path:
PciBindGuard::restore_with_cached_id called deregister_vfio_id_by_value, but
that path wrote remove_id directly and bypassed the in-memory refcount used by
deregister_vfio_new_id. If two devices share the same vendor:device ID, a
cached release can remove the kernel match-table ID without decrementing the
process-local count. The next normal release may then think another user still
exists and skip the final remove_id write.

The proposed commit makes both BDF-based and cached-ID deregistration go through
the same registry path. It also makes the registry operations explicit:
register/deregister mutate the process-local state and return a small enum
describing whether the sysfs write is needed.

Separately, Codex pointed out that another rollback boundary in
bind_device_to_vfio looks risky.

The function unbinds the current host driver before writing
driver_override=vfio-pci. If the driver_override write then fails, the
current path deregisters the vfio ID and returns, but it does not re-probe the
device. That can leave the device detached from its host driver.

I think this is worth treating as a sysfs state-transition boundary rather than
only a one-line error-path issue. The bind flow mutates several kernel-facing
bits independently: current driver binding, vfio-pci new_id,
driver_override, and drivers_probe. Rollback probably needs to track which
mutations this call actually performed. For example, if this call unbound the
host driver, later failures should best-effort re-probe the device; clearing
driver_override should be reserved for paths where the override write may have
succeeded.

A representative regression test would be:

#[test]
fn test_bind_failure_reprobes_after_driver_override_failure() {
    let _refcount_guard = test_refcounts::guard();
    test_refcounts::clear("10de 26bb");
    let (tmp, sysfs) = setup_mock_sysfs();
    let bdf = "0000:2d:00.0";

    create_pci_device(&sysfs, tmp.path(), bdf, "0x10de", "0x26bb", "0x030000", 42);
    create_new_id_file(&sysfs);
    create_remove_id_file(&sysfs);
    create_probe_file(&sysfs);
    set_mock_driver(&sysfs, bdf, "nvidia");

    // Force driver_override to fail after the host driver has been unbound.
    fs::create_dir(sysfs.pci_device(bdf).join("driver_override")).unwrap();

    let err = bind_device_to_vfio(&sysfs, bdf).unwrap_err();
    assert!(matches!(err, VfioError::BindFailed { .. }));

    let probed = fs::read_to_string(sysfs.drivers_probe()).unwrap();
    assert_eq!(
        probed, bdf,
        "driver_override failure happens after unbind, so rollback should re-probe the host driver"
    );
}

On the current PR, this fails because drivers_probe is never written on the
driver_override failure path.

elezar

Thanks @cheese-head. Feel free to drop the commit I suggested if you think it doesn't go in the right direction.

cheese-head · 2026-05-28T16:17:08Z

+    let override_path = sysfs.pci_device_ref(bdf).driver_override_path();
+    if let Err(e) = write_sysfs(&override_path, "vfio-pci") {
+        deregister_vfio_new_id(sysfs, bdf);
+        return Err(VfioError::BindFailed {


should we also call cleanup_partial_bind for this error branch as well?

generalize crate for multi-device PCIe passthrough

46f40b4

Signed-off-by: Patrick Riel <priel@nvidia.com>

cheese-head requested review from a team, derekwaynecarr, maxamillion and mrunalp as code owners May 26, 2026 18:26

drew requested a review from elezar May 26, 2026 20:58

add adopt apis which allow for devices already bound to vfio-pci duri…

b3f4e62

…ng restart reconciliation, without rebinding or mutating sysfs. Signed-off-by: Patrick Riel <priel@nvidia.com>

elezar reviewed May 27, 2026

View reviewed changes

Comment thread crates/openshell-vfio/src/bind.rs

cheese-head and others added 2 commits May 27, 2026 16:27

refactor(vfio): generalize GPU passthrough sysfs handling

cc22198

Signed-off-by: Patrick Riel <priel@nvidia.com>

fix(vfio): centralize vfio ID refcounting

656eaf2

Signed-off-by: Evan Lezar <elezar@nvidia.com>

elezar force-pushed the feat/vfio-multi-device-passthrough branch from a112488 to 656eaf2 Compare May 28, 2026 12:06

elezar reviewed May 28, 2026

View reviewed changes

cheese-head commented May 28, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

generalize crate for multi-device PCIe passthrough#1573

generalize crate for multi-device PCIe passthrough#1573
cheese-head wants to merge 4 commits into
NVIDIA:mainfrom
cheese-head:feat/vfio-multi-device-passthrough

cheese-head commented May 26, 2026

Uh oh!

copy-pr-bot Bot commented May 26, 2026

Uh oh!

drew commented May 26, 2026

Uh oh!

elezar May 27, 2026

Uh oh!

Uh oh!

elezar commented May 28, 2026

Uh oh!

elezar left a comment

Uh oh!

cheese-head May 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

cheese-head commented May 26, 2026

Summary

Related Issue

Changes

Testing

Checklist

Uh oh!

copy-pr-bot Bot commented May 26, 2026

Uh oh!

drew commented May 26, 2026

Uh oh!

elezar May 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elezar commented May 28, 2026

Uh oh!

elezar left a comment

Choose a reason for hiding this comment

Uh oh!

cheese-head May 28, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants